Skip to content

Implement MemoryAnalyticsService for semantic layer queries#625

Merged
hotlong merged 3 commits into
mainfrom
copilot/evaluate-memory-driver-analytics
Feb 11, 2026
Merged

Implement MemoryAnalyticsService for semantic layer queries#625
hotlong merged 3 commits into
mainfrom
copilot/evaluate-memory-driver-analytics

Conversation

Copilot AI commented Feb 11, 2026

Copy link
Copy Markdown
Contributor

InMemoryDriver has full aggregation capabilities via Mingo but IAnalyticsService only had a stub returning empty results. This implements a production-ready analytics service for development/testing.

Implementation

New: MemoryAnalyticsService (packages/plugins/driver-memory/src/memory-analytics.ts)

  • Implements IAnalyticsService using InMemoryDriver's MongoDB-compatible aggregation pipeline
  • Translates cube-based semantic queries to Mingo aggregation stages
  • Handles field name mapping (MongoDB doesn't support dots in $project keys)

Schema Alignment

  • Added order and timezone fields to AnalyticsQuery interface
  • Synchronized Zod schema with TypeScript contract
  • Fixed cube field reference in AnalyticsQuerySchema

Usage

import { MemoryAnalyticsService, InMemoryDriver } from '@objectstack/driver-memory';

const driver = new InMemoryDriver({ initialData: { orders: [...] } });
await driver.connect();

const service = new MemoryAnalyticsService({
  driver,
  cubes: [{
    name: 'orders',
    sql: 'orders',
    measures: {
      count: { type: 'count', sql: 'id' },
      revenue: { type: 'sum', sql: 'amount' }
    },
    dimensions: {
      status: { type: 'string', sql: 'status' }
    }
  }]
});

const result = await service.query({
  cube: 'orders',
  measures: ['orders.revenue'],
  dimensions: ['orders.status'],
  order: { 'orders.revenue': 'desc' }
});

Technical Notes

MongoDB aggregation creates nested objects for dotted field names in $project. Pipeline uses short names internally, then maps to cube.field format in final results.

Test Coverage: 18 tests covering metadata discovery, aggregations (count/sum/avg/min/max), grouping, filters, sorting, pagination, SQL generation.

Original prompt

评估目前的 memory driver 应该已经有数据分析能力,是否应实现 数据分析服务。

The user has attached the following file paths as relevant context:

  • .github/copilot-instructions.md

Created from VS Code.


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

@vercel

vercel Bot commented Feb 11, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
objectstack-play Ready Ready Preview, Comment Feb 11, 2026 5:05pm
spec Error Error Feb 11, 2026 5:05pm

Request Review

Copilot AI and others added 2 commits February 11, 2026 15:34
- Created MemoryAnalyticsService using InMemoryDriver's aggregation capabilities
- Added comprehensive test suite for analytics service
- Updated analytics contract interface to include order and timezone fields
- Aligned Zod schema with TypeScript interface
- Exported MemoryAnalyticsService from driver-memory package

Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>
- Fixed field naming issue in aggregation pipeline
- Use short names internally and map to full cube.field names in results
- Fixed sorting to use short field names
- Added driver.connect() call in tests to load initial data
- All 18 tests now passing

Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>
Copilot AI changed the title [WIP] Assess implementation of data analysis service for memory driver Implement MemoryAnalyticsService for semantic layer queries Feb 11, 2026
Copilot AI requested a review from hotlong February 11, 2026 15:41
@hotlong

hotlong commented Feb 11, 2026

Copy link
Copy Markdown
Collaborator

@copilot 继续

@hotlong
hotlong marked this pull request as ready for review February 11, 2026 15:54
Copilot AI review requested due to automatic review settings February 11, 2026 15:54
@hotlong
hotlong merged commit 187e7ae into main Feb 11, 2026
3 of 5 checks passed
Copilot AI requested a review from hotlong February 11, 2026 15:55
Copilot stopped work on behalf of hotlong due to an error February 11, 2026 15:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements a functional in-memory analytics service that translates semantic-layer cube queries into MongoDB/Mingo-style aggregation pipelines, replacing a previously stubbed analytics path and aligning query schemas/contracts.

Changes:

  • Added MemoryAnalyticsService with aggregation, filtering, grouping, sorting, pagination, and SQL/pipeline debug output.
  • Updated analytics query contracts/schemas to include cube, order, and timezone alignment changes.
  • Added Vitest coverage for metadata discovery and common aggregations/operations.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
packages/spec/src/data/analytics.zod.ts Updates AnalyticsQuerySchema (adds cube, adjusts timezone typing/default).
packages/spec/src/contracts/analytics-service.ts Extends AnalyticsQuery with order and timezone.
packages/spec/src/api/analytics.zod.ts Moves cube into query by removing top-level cube from request schema.
packages/plugins/driver-memory/src/memory-driver.ts Refactors imports to import type for type-only dependencies.
packages/plugins/driver-memory/src/memory-analytics.ts New in-memory analytics implementation using aggregation pipelines + helpers.
packages/plugins/driver-memory/src/memory-analytics.test.ts Adds tests for meta, aggregations, grouping, filters, sorting, pagination, SQL output.
packages/plugins/driver-memory/src/index.ts Exports MemoryAnalyticsService and its config type.

Comment on lines +374 to +393
private buildAggregator(measure: { type: string; sql: string; filters?: any[] }): any {
const fieldPath = measure.sql.replace(/^\$/, '');

switch (measure.type) {
case 'count':
return { $sum: 1 };
case 'sum':
return { $sum: `$${fieldPath}` };
case 'avg':
return { $avg: `$${fieldPath}` };
case 'min':
return { $min: `$${fieldPath}` };
case 'max':
return { $max: `$${fieldPath}` };
case 'count_distinct':
return { $addToSet: `$${fieldPath}` }; // Will need post-processing for count
default:
return { $sum: 1 }; // Default to count
}
}

Copilot AI Feb 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

count_distinct currently returns an array (from $addToSet) and is never converted to a numeric count in the pipeline, so query results will be incorrect. A concrete fix is to detect count_distinct measures and, in the $project stage, project { $size: '$<measureName>' } (or add an additional projection stage) so the final output is a number.

Copilot uses AI. Check for mistakes.
Comment on lines +68 to +89
if (query.filters && query.filters.length > 0) {
const matchStage: Record<string, any> = {};
for (const filter of query.filters) {
const mongoOp = this.convertOperatorToMongo(filter.operator);
const fieldPath = this.resolveFieldPath(cube, filter.member);

if (filter.values && filter.values.length > 0) {
if (mongoOp === '$in') {
matchStage[fieldPath] = { $in: filter.values };
} else if (mongoOp === '$nin') {
matchStage[fieldPath] = { $nin: filter.values };
} else {
matchStage[fieldPath] = { [mongoOp]: filter.values[0] };
}
} else if (mongoOp === '$exists') {
matchStage[fieldPath] = { $exists: filter.operator === 'set' };
}
}
if (Object.keys(matchStage).length > 0) {
pipeline.push({ $match: matchStage });
}
}

Copilot AI Feb 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Several operators are mapped to Mongo operators that won’t work with the way $match is built here: contains is mapped to $regex but passed a raw string (often needs a RegExp or a { $regex, $options } pattern), notContains maps to $not but $not expects an operator expression (commonly { $not: { $regex: ... } }), and inDateRange maps to $gte but should apply both $gte and $lte on the same field. Consider handling these operators explicitly when building matchStage instead of relying on convertOperatorToMongo alone.

Copilot uses AI. Check for mistakes.
Comment on lines +170 to +173
if (query.offset) {
pipeline.push({ $skip: query.offset });
}
if (query.limit) {

Copilot AI Feb 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using truthy checks here causes offset: 0 and limit: 0 to be ignored (even if the caller explicitly sets them). Use an explicit query.offset !== undefined / query.limit !== undefined (or != null) check so zero values are respected.

Suggested change
if (query.offset) {
pipeline.push({ $skip: query.offset });
}
if (query.limit) {
if (query.offset !== undefined) {
pipeline.push({ $skip: query.offset });
}
if (query.limit !== undefined) {

Copilot uses AI. Check for mistakes.
Comment on lines +69 to +87
const matchStage: Record<string, any> = {};
for (const filter of query.filters) {
const mongoOp = this.convertOperatorToMongo(filter.operator);
const fieldPath = this.resolveFieldPath(cube, filter.member);

if (filter.values && filter.values.length > 0) {
if (mongoOp === '$in') {
matchStage[fieldPath] = { $in: filter.values };
} else if (mongoOp === '$nin') {
matchStage[fieldPath] = { $nin: filter.values };
} else {
matchStage[fieldPath] = { [mongoOp]: filter.values[0] };
}
} else if (mongoOp === '$exists') {
matchStage[fieldPath] = { $exists: filter.operator === 'set' };
}
}
if (Object.keys(matchStage).length > 0) {
pipeline.push({ $match: matchStage });

Copilot AI Feb 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there are multiple filters targeting the same fieldPath, this logic overwrites the previous condition (last filter wins), which will produce incorrect filtering. A concrete approach is to build an $and array of per-filter expressions (or merge operators onto the same field where safe) and emit $match: { $and: [...] } when needed.

Suggested change
const matchStage: Record<string, any> = {};
for (const filter of query.filters) {
const mongoOp = this.convertOperatorToMongo(filter.operator);
const fieldPath = this.resolveFieldPath(cube, filter.member);
if (filter.values && filter.values.length > 0) {
if (mongoOp === '$in') {
matchStage[fieldPath] = { $in: filter.values };
} else if (mongoOp === '$nin') {
matchStage[fieldPath] = { $nin: filter.values };
} else {
matchStage[fieldPath] = { [mongoOp]: filter.values[0] };
}
} else if (mongoOp === '$exists') {
matchStage[fieldPath] = { $exists: filter.operator === 'set' };
}
}
if (Object.keys(matchStage).length > 0) {
pipeline.push({ $match: matchStage });
const andConditions: Record<string, any>[] = [];
for (const filter of query.filters) {
const mongoOp = this.convertOperatorToMongo(filter.operator);
const fieldPath = this.resolveFieldPath(cube, filter.member);
const condition: Record<string, any> = {};
if (filter.values && filter.values.length > 0) {
if (mongoOp === '$in') {
condition[fieldPath] = { $in: filter.values };
} else if (mongoOp === '$nin') {
condition[fieldPath] = { $nin: filter.values };
} else {
condition[fieldPath] = { [mongoOp]: filter.values[0] };
}
} else if (mongoOp === '$exists') {
condition[fieldPath] = { $exists: filter.operator === 'set' };
}
if (Object.keys(condition).length > 0) {
andConditions.push(condition);
}
}
if (andConditions.length === 1) {
pipeline.push({ $match: andConditions[0] });
} else if (andConditions.length > 1) {
pipeline.push({ $match: { $and: andConditions } });

Copilot uses AI. Check for mistakes.
Comment on lines +69 to +87
const matchStage: Record<string, any> = {};
for (const filter of query.filters) {
const mongoOp = this.convertOperatorToMongo(filter.operator);
const fieldPath = this.resolveFieldPath(cube, filter.member);

if (filter.values && filter.values.length > 0) {
if (mongoOp === '$in') {
matchStage[fieldPath] = { $in: filter.values };
} else if (mongoOp === '$nin') {
matchStage[fieldPath] = { $nin: filter.values };
} else {
matchStage[fieldPath] = { [mongoOp]: filter.values[0] };
}
} else if (mongoOp === '$exists') {
matchStage[fieldPath] = { $exists: filter.operator === 'set' };
}
}
if (Object.keys(matchStage).length > 0) {
pipeline.push({ $match: matchStage });

Copilot AI Feb 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there are multiple filters targeting the same fieldPath, this logic overwrites the previous condition (last filter wins), which will produce incorrect filtering. A concrete approach is to build an $and array of per-filter expressions (or merge operators onto the same field where safe) and emit $match: { $and: [...] } when needed.

Suggested change
const matchStage: Record<string, any> = {};
for (const filter of query.filters) {
const mongoOp = this.convertOperatorToMongo(filter.operator);
const fieldPath = this.resolveFieldPath(cube, filter.member);
if (filter.values && filter.values.length > 0) {
if (mongoOp === '$in') {
matchStage[fieldPath] = { $in: filter.values };
} else if (mongoOp === '$nin') {
matchStage[fieldPath] = { $nin: filter.values };
} else {
matchStage[fieldPath] = { [mongoOp]: filter.values[0] };
}
} else if (mongoOp === '$exists') {
matchStage[fieldPath] = { $exists: filter.operator === 'set' };
}
}
if (Object.keys(matchStage).length > 0) {
pipeline.push({ $match: matchStage });
const matchConditions: Record<string, any>[] = [];
for (const filter of query.filters) {
const mongoOp = this.convertOperatorToMongo(filter.operator);
const fieldPath = this.resolveFieldPath(cube, filter.member);
let fieldCondition: Record<string, any> | null = null;
if (filter.values && filter.values.length > 0) {
if (mongoOp === '$in') {
fieldCondition = { [fieldPath]: { $in: filter.values } };
} else if (mongoOp === '$nin') {
fieldCondition = { [fieldPath]: { $nin: filter.values } };
} else {
fieldCondition = { [fieldPath]: { [mongoOp]: filter.values[0] } };
}
} else if (mongoOp === '$exists') {
fieldCondition = { [fieldPath]: { $exists: filter.operator === 'set' } };
}
if (fieldCondition) {
matchConditions.push(fieldCondition);
}
}
if (matchConditions.length === 1) {
pipeline.push({ $match: matchConditions[0] });
} else if (matchConditions.length > 1) {
pipeline.push({ $match: { $and: matchConditions } });

Copilot uses AI. Check for mistakes.
Comment on lines +91 to +112
// Stage 2: Time dimension filters
if (query.timeDimensions && query.timeDimensions.length > 0) {
for (const timeDim of query.timeDimensions) {
const fieldPath = this.resolveFieldPath(cube, timeDim.dimension);
if (timeDim.dateRange) {
const range = Array.isArray(timeDim.dateRange)
? timeDim.dateRange
: this.parseDateRangeString(timeDim.dateRange);

if (range.length === 2) {
pipeline.push({
$match: {
[fieldPath]: {
$gte: new Date(range[0]),
$lte: new Date(range[1])
}
}
});
}
}
}
}

Copilot AI Feb 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

timeDimensions includes granularity, but this implementation only applies a date range filter and does not transform/group by time buckets. If callers expect results grouped by day/week/month via timeDimensions.granularity, the current output will be wrong. Consider incorporating time dimensions into the $group._id (and projected fields) by adding a computed bucket expression (e.g., date truncation / formatted date key) when granularity is provided.

Copilot uses AI. Check for mistakes.
Comment on lines 135 to 138
export const AnalyticsQuerySchema = z.object({
cube: z.string().describe('Target cube name'),
measures: z.array(z.string()).describe('List of metrics to calculate'),
dimensions: z.array(z.string()).optional().describe('List of dimensions to group by'),

Copilot AI Feb 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AnalyticsQuery (TypeScript contract) now includes order?: Record<string, 'asc' | 'desc'>, but AnalyticsQuerySchema does not define order. This can cause runtime validation to reject otherwise-valid requests (or strip the field, depending on schema settings). Add order to the Zod schema to keep runtime validation aligned with the contract.

Copilot uses AI. Check for mistakes.
Comment on lines +160 to 161
timezone: z.string().optional().default('UTC'),
});

Copilot AI Feb 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AnalyticsQuery (TypeScript contract) now includes order?: Record<string, 'asc' | 'desc'>, but AnalyticsQuerySchema does not define order. This can cause runtime validation to reject otherwise-valid requests (or strip the field, depending on schema settings). Add order to the Zod schema to keep runtime validation aligned with the contract.

Copilot uses AI. Check for mistakes.
@@ -29,8 +29,7 @@ export const AnalyticsEndpoint = z.enum([
* Query Request Body
*/
export const AnalyticsQueryRequestSchema = z.object({

Copilot AI Feb 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change removes the top-level cube field from the request body schema (previously cube was a sibling of query). If this endpoint is already consumed externally, this is a breaking request-shape change. Consider supporting both shapes during a deprecation window (e.g., accept cube at the top level and/or inside query, then normalize), or ensure versioning/migration guidance is in place.

Suggested change
export const AnalyticsQueryRequestSchema = z.object({
export const AnalyticsQueryRequestSchema = z.object({
/**
* Optional top-level cube selector for backward compatibility.
* Prefer specifying the cube inside `query` instead (e.g., `query.cube`).
*/
cube: z.string().optional().describe('Deprecated: use query.cube instead'),

Copilot uses AI. Check for mistakes.
Comment on lines +248 to +249
expect(result.rows[1]['orders.customer']).toBe('Alice'); // 250
expect(result.rows[2]['orders.customer']).toBe('Bob'); // 250

Copilot AI Feb 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test asserts a deterministic ordering for tied sort keys (Alice and Bob both have totalAmount 250). Since tie-breaking order isn’t guaranteed unless explicitly specified, this can be flaky across implementations/versions. Make the expectation order-insensitive for ties, or add a secondary sort in order (e.g., also sort by orders.customer).

Suggested change
expect(result.rows[1]['orders.customer']).toBe('Alice'); // 250
expect(result.rows[2]['orders.customer']).toBe('Bob'); // 250
const tiedCustomers = [
result.rows[1]['orders.customer'],
result.rows[2]['orders.customer']
].sort();
expect(tiedCustomers).toEqual(['Alice', 'Bob']); // 250, 250 in any order

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants